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Program Description 1 

Saxon Math is a core curriculum for students in grades K-12 that 
uses an incremental approach to instruction and assessment. This 
approach limits the amount of new math content delivered to stu- 
dents each day and allows time for daily practice. New concepts are 
introduced gradually and integrated with previously introduced con- 
tent so that concepts are developed, reviewed, and practiced over 
time rather than being taught during discrete periods of time, such as 
in chapters or units. This review focuses on studies of Saxon Math’s 
secondary courses, including Saxon Algebra I, Saxon Geometry, 

Saxon Algebra II, and Saxon Advanced Math. 

Research 2 

The What Works Clearinghouse (WWC) identified two studies of 
Saxon Algebra I that both fall within the scope of the Secondary 
Mathematics topic area and meet WWC group design standards. 

One study meets WWC group design standards without reservations 
and the other study meets WWC group design standards with reser- 
vations. Together, these studies included 198 secondary students in 
grades 8-9 in two locations. 

The WWC considers the extent of evidence for Saxon Algebra I on the 
mathematics achievement of secondary students to be small for the 
algebra domain. There were no studies that meet WWC group design 
standards in the five other domains, so this intervention report does not report on the effectiveness of Saxon Math for 
those domains. (See the Effectiveness Summary on p. 5 for more details of effectiveness by course and domain.) 

The findings in this report pertain to Saxon Algebra I only. No studies of Saxon Geometry, Saxon Algebra II, or 
Saxon Advanced Math fall within the scope of the Secondary Mathematics review protocol and meet WWC group 
design standards. 

Effectiveness 

Saxon Algebra I was found to have no discernible effects on algebra for secondary students. 
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Table 1. Summary of findings 3 


Improvement index 

(percentile points) 


Course and outcome 
domain 

Rating of 
effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Saxon Algebra 1 

Algebra 

No discernible 
effects 

+5 

+4 to +6 

2 

198 

Small 

Saxon Geometry 

No evidence 






Saxon Algebra II 

No evidence 






Saxon Advanced Math 

No evidence 
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Program Information 

Background 

Originally developed by John Saxon, Saxon Math is distributed by Houghton Mifflin Harcourt Supplemental Publishers. 
Address: Specialized Curriculum Group, 9205 Southpark Center Loop, Orlando, FL, 32819. Email: greatservice@ 
hmhpub.com. Website: www.saxonpublishing.com. Telephone: (800) 289-4490. Fax: (800) 289-3994. 

Program details 

Saxon Math consists of at least 120 daily lessons and 12 activity-based investigations for each subject and grade 
level. Each lesson makes use of three strategies: 

• Each daily lesson consists of learning a new mathematical concept, working on practice problems relating to 
that concept, and solving a number of problems that include the current and previous material. 

• A relatively small set of new math ideas is introduced daily using examples, mathematical conversations, and 
practice, such that new ideas and concepts are integrated with ones that were previously introduced. 

• The lessons include written practice that aims to help students both master new skills and maintain their mas- 
tery of concepts previously instructed. 

Students complete written, cumulative assessments after every five lessons. The results of these assessments pro- 
vide teachers with data for instructional decision making and provide feedback for students and parents. In addi- 
tion to these written assessments, students may demonstrate mastery of math content through alternate interactive 
opportunities, such as investigations, test-day activities, and performance tasks. 

The secondary curriculum includes Saxon Algebra I, Saxon Geometry, Saxon Algebra II, and Saxon Advanced Math, 
which can be purchased separately. 


Cost 

As of November 2015, the student editions of Saxon Algebra I, Saxon Algebra II, Saxon Geometry, and Saxon 
Advanced Math cost $82.90, $86.45, $93.60, and $89.95, respectively. The teacher editions of each subject cost 
$84.30, $92.20, $1 31 .45, and $91 .35, respectively. Other curriculum materials include eBooks, student work- 
books, and teaching materials, and range in price from $10.85 for a Saxon Algebra I Student Practice Workbook to 
$2,244.15 for 6-year Saxon Algebra I online edition with Destination Math. 
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Research Summary 

This research summary includes information from studies of all avail- Table 2. Scope of reviewed research 
able Saxon Math courses for secondary students. 

The WWC identified 1 1 eligible studies for review: 

• Ten eligible studies investigated the effects of Saxon Algebra I on 
the mathematics achievement of secondary students. 

• One eligible study investigated the cumulative effect of 3 years of Saxon Math courses (including Saxon Alge- 
bra I, Saxon Algebra II, and Saxon Advanced Mathematics) on the mathematics achievement of secondary 
students. 4 

The WWC reviewed the 1 1 eligible studies against group design standards. One study (Pierce, 1984) is a random- 
ized controlled trial that meets WWC group design standards without reservations, and one study (Peters, 1992) 
is a randomized controlled trial that meets WWC group design standards with reservations. Those two studies are 
summarized in this report. The remaining nine studies do not meet WWC group design standards. 

An additional 46 studies were identified but do not meet WWC eligibility criteria for review in this topic area. Cita- 
tions for all 57 studies are in the References section, which begins on p. 6. 

Summary of Saxon Algebra / study meeting WWC group design standards without reservations 

Pierce (1984) conducted a randomized controlled trial in one high school in the southern midwest United States. 

The study sample included ninth-grade students enrolled in beginning algebra. Two eligible teachers were randomly 
selected to participate in the study. For each teacher, half of their classes were randomly assigned to receive Saxon 
Algebra I, while the other half were randomly assigned to receive the business-as-usual algebra curriculum ( Holt 
Algebra /). While the study author refers to the study as using a quasi-experimental design, this random assign- 
ment of classes led the WWC to review the study as a randomized controlled trial. Students were assigned via the 
school’s normal computerized scheduling procedures. The author notes that students who were assigned to begin- 
ning algebra after the third week of school were assigned to one of the 12 other classrooms not in the study. The 
final sample included 162 students (77 intervention and 85 comparison). 

Summary of Saxon Algebra / study meeting WWC group design standards with reservations 

Peters (1 992) conducted a randomized controlled trial that investigated the effect of Saxon Algebra I on the mathe- 
matics achievement of 36 “math-talented” eighth-grade students (19 Saxon Math and 17 comparison) from one junior 
high school in Nebraska during the 1991-92 school year. 5 The district borders two large cities (Lincoln and Omaha), 
and its students lived in rural and suburban areas. Students in the intervention group used Saxon Algebra I (1981), 
while students in the comparison group used the University of Chicago School Mathematics Project (UCSMP) 

Algebra I First-Edition textbook. The integrity of random assignment was compromised because some students did 
not remain in the study group to which they were randomly assigned— students were reallocated between the inter- 
vention and comparison groups to accommodate scheduling difficulties and student requests for other course offer- 
ings. The study demonstrated baseline equivalence on the analysis sample and therefore meets WWC group design 
standards with reservations. 


Grade 

8-12 

Delivery method 

Whole class 

Program type 

Curriculum 
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Effectiveness Summary 

The WWC review of Saxon Algebra I for the Secondary Mathematics topic area includes student outcomes in six 
domains: algebra, geometry, statistics and probability, trigonometry/precalculus, calculus, and general mathemat- 
ics achievement. The two studies of Saxon Algebra I that meet WWC group design standards reported findings in 
one of the six domains: algebra. The findings below present the authors’ estimates and WWC-calculated estimates 
of the size and statistical significance of the effects of Saxon Algebra I on secondary students. Additional com- 
parisons are presented as supplemental findings in the appendix. The supplemental findings do not factor into the 
intervention’s rating of effectiveness. For a more detailed description of the rating of effectiveness and extent of 
evidence criteria, see the WWC Rating Criteria on p. 19. 


Summary of Saxon Algebra / effectiveness for the algebra domain 

Table 3. Rating of effectiveness of Saxon Algebra I and extent of evidence for the algebra domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

No evidence of statistically 
significant or substantively 
important effects, either 
positive or negative. 

In the two studies that reported findings, the estimated impact of the intervention on outcomes in the algebra 
domain was neither statistically significant nor large enough to be considered substantively important. 

Extent of evidence 

Criteria met 

Small 

Two studies that included 198 students in two schools reported evidence of effectiveness in the algebra domain. 


Two studies of Saxon Algebra I that meet WWC group design standards with or without reservations reported 
findings in the algebra domain. 

Pierce (1984) found, and the WWC confirmed, no statistically significant or substantively important differences 
between the Saxon Algebra I and comparison groups in the algebra domain. The WWC characterizes these study 
findings as an indeterminate effect. 

Peters (1992) found, and the WWC confirmed, no statistically significant or substantively important differences 
between the Saxon Algebra I and comparison groups in the algebra domain. The WWC characterizes these study 
findings as an indeterminate effect. 

Thus, for the algebra domain, two studies of Saxon Algebra I showed indeterminate effects. This results in a rating 
of no discernible effects, with a small extent of evidence. 
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Appendix A.1: Research details for Pierce (1984) 

Pierce, R. D. (1984). A quasi-experimental study of Saxon’s Incremental Development Model and its 
effects on student achievement in first-year algebra (Unpublished doctoral dissertation). Univer- 
sity of Tulsa, OK. 

Table A1. Summary of Saxon Algebra / findings Meets WWC group design standards without reservations 


Study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Outcome domain 

Study findings 

Average improvement index 

Sample size (percentile points) Statistically significant 

Algebra 

162 students +4 No 

Setting 

The study was conducted in a high school in the southern midwest United States. 

Study sample 

The study sample included ninth-grade students enrolled in beginning algebra. Prior to the 
start of the school year, teachers with at least 3 years of experience and an even number of 
ninth-grade beginning algebra classes were identified. Two of the teachers who agreed to 
participate were randomly selected to take part in the study. One teacher taught four classes 
of beginning algebra, and the other taught two classes. For each teacher, half of their classes 
were randomly assigned to receive Saxon Math, while the other half were randomly assigned 
to receive the business-as-usual algebra curriculum (Holt Algebra /). Participating students 
were assigned via the schools’ normal computerized class scheduling procedures (82 inter- 
vention students and 92 comparison). Students who were assigned to beginning algebra after 
the third week of school were assigned to one of 12 non-study classrooms. Analytic sample 
sizes for the unadjusted analysis were 77 intervention students and 85 comparison students. 

The school district was described as primarily White and middle or upper-middle class. Ninth 
graders enrolled in beginning algebra were on the average mathematics track and constituted 
the majority of ninth graders at the study school. 

Intervention 

group 

Students in the intervention group received algebra instruction using Saxon Algebra 1. 

Instruction in both conditions occurred over the course of an entire academic year during 
daily 55-minute math instructional blocks. Teachers organized their classroom instruction in 
“equivalent” ways for both the intervention and comparison sections, with respect to structure: 
utilizing 10-15 minutes for homework review, 10 minutes for review of the text, 15 minutes 
to review new material, and 15-20 minutes of work on problems due during the next class 
period. Class pacing was similar between the two groups; however, instructional approaches 
and problem set content were different based on the differential approaches of the intervention 
and comparison curricula. For example, students in the intervention group received ongoing 
review in accordance with Saxon’s Incremental Development Model. 

Comparison 

group 

Students in the comparison group received instruction using Holt Algebra /, the school’s busi- 
ness-as-usual algebra curriculum. This curriculum followed standard mathematical pedagogy, 
including chapter reviews and practice problems, with problem sets and quarterly review tests. 
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Outcomes and 
measurement 


Support for 
implementation 


The primary outcome measure used in the study was the Lankton First-Year Algebra Test, a 
standardized and normed multiple choice algebra assessment. The assessment was admin- 
istered at the conclusion of the school year. The author also reported subscales defined by 
their alignment with instructional objectives and instructional content, which were defined as 
part of the Lankton assessment. These scores are presented in Appendix D. All scores on 
the Lankton measures are reported as the raw number of test items answered correctly. The 
author presented supplemental findings for several subscores on the assessment. The supple- 
mental findings do not factor into the intervention’s rating of effectiveness. For a more detailed 
description of the outcome measure, see Appendix B. 

The two study teachers met with each other weekly and with the researcher monthly during 
the school year. The meetings were designed as an opportunity to discuss common problems 
encountered implementing Saxon Algebra I and share ideas about the study. Teachers agreed 
at the outset of the study to provide “equivalent” instruction to each study condition to main- 
tain the integrity of the study. 
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Appendix A.2: Research details for Peters (1992) 

Peters, K. G. (1992). Skill performance comparability of two algebra programs on an eighth-grade 
population. Dissertation Abstracts International, 54(01), 77A. (UMI No. 9314428) 

Table A2. Summary of Saxon Algebra / findings Meets WWC group design standards with reservations 


Study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Outcome domain 

Study findings 

Average improvement index 

Sample size (percentile points) Statistically significant 

Algebra 

36 students +6 No 

Setting 

The study took place in one junior high school in Nebraska. The district borders two large 
cities (Lincoln and Omaha) and has a mix of students living in rural and suburban locations. 

Study sample 

The study sample included 36 students from two classrooms taught by the same eighth- 
grade teacher in one junior high school during the 1991-92 school year. All of the students 
were “math-talented” based on teacher recommendations and prior academic achievement. 

No information is provided on the specific thresholds that were used in delineating the math- 
talented criteria; however, all students scored at or above the 87th percentile on the California 
Achievement Test (CAT) Total Math battery. Of the total sample, 56% were female (58% inter- 
vention and 53% comparison) and 44% were male (42% intervention and 47% comparison). 
Students were randomly assigned to the teacher’s two classrooms, and the teacher used the 
intervention curriculum in one classroom and the comparison curriculum in the other. However, 
the assignment of students was altered after random assignment to accommodate scheduling 
difficulties and student requests for other course offerings. The analytic sample included 19 
students in the Saxon Math group and 17 students in the comparison group. 

Intervention 

group 

Students in the intervention group were taught using Saxon Algebra 1 (1981) during the 1991- 
92 school year. Students participated in daily math lessons for one academic year. In each 
lesson, the teacher introduced a new concept, and students had opportunities to practice the 
new concept and past concepts. Students were assessed every fifth lesson with study- 
specific tests of the material covered in the past few sessions. 

Comparison 

group 

Students in the comparison group were taught using the UCSMP Algebra curriculum. The 
UCSMP Algebra program was developed based on National Council of the Teachers of 
Mathematics (NCTM) objectives that emphasized problem-solving skills, reading comprehen- 
sion, use of technology, and relevant lessons with real-world applications. Each lesson is 
organized into an introduction of the concept, a reading section that explains the process, 
and real-life problem situations. 

Outcomes and 
measurement 

The primary outcome measure was the Orleans-Hanna Algebra Prognosis Test. 6 This measure 
was administered as a pretest in August 1991 and as a posttest in May 1992. For the pretest 
measure, the author reported both a standardized score and a raw score. Only the standard- 
ized score is used in this review because, per the author, the standardized score allows com- 
parability between the pretest and posttest. For a more detailed description of the outcome 
measure, see Appendix B. 
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Support for 
implementation 


The teacher who taught both study groups did not have prior experience with the intervention 
or comparison curricula but had read extensively about both instructional approaches. The 
teacher participated in a 1-week summer workshop on UCSMP Algebra, and in two 1-day 
workshops given by local consultants on both curricula used in the study. The study author 
also conducted weekly monitoring to help maintain implementation integrity. 


Saxon Math May 201 6 


Page 14 


WWC Intervention Report 


Appendix B: Outcome measures for the algebra domain 


Algebra 


Algebra 


Lankton First-Year Algebra Test 

The Lankton First-Year Algebra Test is a standardized and normed algebra assessment consisting of 50 multiple-choice 
questions. The test is designed to assess knowledge of the basic objectives and content of a first-year algebra course. 

The objectives include an understanding of the concepts of number, set, operation, structure, and relation. Test items 
are also organized into content categories, including: defining terms, meanings of signs and symbols; algebraic opera- 
tions, factoring and square roots; equations and inequalities; algebraic expressions and formulas, functions, variation, 
problem solving; and graphic representation. Reported reliability for this measure was 0.86 (as cited in Pierce, 1984). 

The author also reports subscales defined by instructional objectives and content. The five content subscales include: 1. 
Definition of terms, meaning of signs and symbols; 2. Fundamental algebraic operations, factoring and extracting roots; 

3. Equations and inequalities; 4. Algebraic expressions and formulas, functions, variation, problem solving; and 5. Graphic 
representation. The five instructional objectives include: 1. Number; 2. Set; 3. Operation; 4. Structure; and 5. Relation. 

The subscores are reported as supplemental findings in Appendix D. 

Orleans- Hanna Algebra 
Prognosis Test 

The 60-item nationally normed Orleans-Hanna Algebra Prognosis Test is used to place and group students in algebra 
courses and inform the development of lessons plans. It was developed in 1928 and revised in 1980. This measure was 
administered as a pretest in August 1991 and as a posttest in May 1992. For the pretest measure, the author reports 
both a standardized score and a raw score. Only the standardized score is used in this review because, per the author, 
the standardized score allows comparability between the pretest and posttest. The reported sample Kuder-Richardson 
reliability estimate was .96 (as cited in Peters, 1992). 
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Appendix C: Findings included in the rating for studies of Saxon Algebra I for the algebra domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Pierce (1984) a 

Lankton First-Year Algebra Test 

Grade 9 

162 

students 

22.88 

(6.34) 

22.34 

(5.48) 

0.54 

0.09 

+4 

.44 

Domain average for algebra (Pierce, 1984) 





0.09 

+4 

Not 

statistically 

significant 

Peters (1992) b 

Orleans-Hanna Algebra 

Prognosis Test 

Grade 8 
(math- 

36 

Students 

95.63 

(4.53) 

95.06 

(4.09) 

0.61 

0.14 

+6 

>.05 


talented) 


Domain average for algebra (Peters, 1992) 0.14 +6 Not 

statistically 

significant 


Domain average for algebra across all studies 0.11 +5 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who 
are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change 
in an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to 
two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of each study's domain average was determined by the 
WWC. Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Pierce (1 984), a correction for clustering was needed but did not affect whether any of the contrasts were found to be statistically significant. The p-value presented here was 
reported in the original study. The author also conducted an ANCOVA analysis that adjusted for student pretest scores on a measure of mathematical knowledge. However, due to 
missing data in the ANCOVA analysis, the WWC focuses on the unadjusted analysis reported in the study. Reported p-values are similar across the two analyses. Because pretest 
standard deviations were not reported, calculation of the program group mean using a difference-in-differences approach was not feasible. This study is characterized as having an 
indeterminate effect because the estimated effect is neither statistically significant nor substantively important. For more information, please refer to the WWC Standards and 
Procedures Handbook (version 3.0), p. 26. 

b For Peters (1 992), the WWC did not need to make corrections for clustering, multiple comparisons, or to adjust for baseline differences. The WWC calculated the program group mean 
using a difference-in-differences approach (see WWC Procedures and Standards Handbook) by adding the impact of the program (i.e., difference in mean gains between the interven- 
tion and comparison groups) to the unadjusted comparison group posttest means. Please see the WWC Procedures and Standards Handbook (version 3.0) for more information. The 
p-value presented here was reported in the original study. This study is characterized as having an indeterminate effect because the estimated effect is neither statistically significant 
nor substantively important. For more information, please refer to the WWC Standards and Procedures Handbook (version 3.0), p. 26. 
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Appendix D: Description of supplemental findings for studies of Saxon Algebra I for the algebra domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Pierce (1984) a 

Lankton First-Year Algebra Test: 
Content Subscale 1 -Defining terms, 
meanings of signs and symbols 

Grade 9 

162 

students 

4.23 

(1.64) 

3.84 

(1.58) 

0.39 

0.24 

+10 

.21 

Lankton First-Year Algebra Test: 
Content Subscale 2-Algebraic 
operations, factoring and square 
roots 

Grade 9 

162 

students 

5.47 

(2.08) 

5.97 

(1.79) 

-0.50 

-0.26 

-10 

.15 

Lankton First-Year Algebra Test: 
Content Subscale 3-Equations and 
inequalities 

Grade 9 

162 

students 

6.29 

(2.19) 

6.04 

(2.08) 

0.25 

0.12 

+5 

.37 

Lankton First-Year Algebra Test: 
Content Subscale 4-Algebraic 
expressions and formulas, functions, 
variation, problem solving 

Grade 9 

162 

students 

3.53 

(1.73) 

3.53 

(1.59) 

0.00 

0.00 

0 

.98 

Lankton First-Year Algebra Test: 
Content Subscale 5-Graphic 
representation 

Grade 9 

162 

students 

3.38 

(1.62) 

2.93 

(1.62) 

0.45 

0.28 

+11 

.04 

Lankton First-Year Algebra Test: 
Instructional Subscale 1 -Number 

Grade 9 

162 

students 

6.27 

(1.80) 

6.13 

(2.01) 

0.14 

0.07 

+3 

.44 

Lankton First-Year Algebra Test: 
Instructional Subscale 2-Set 

Grade 9 

162 

students 

3.78 

(1.61) 

3.58 

(1.64) 

0.20 

0.12 

+5 

.30 

Lankton First-Year Algebra Test: 
Instructional Subscale 3-Operation 

Grade 9 

162 

students 

4.84 

(1.92) 

5.02 

(1.55) 

-0.18 

-0.10 

-4 

.50 

Lankton First-Year Algebra Test: 
Instructional Subscale 4-Structure 

Grade 9 

162 

students 

5.26 

(2.05) 

4.87 

(1.84) 

0.39 

0.20 

+8 

.17 

Lankton First-Year Algebra Test: 
Instructional Subscale 5-Relation 

Grade 9 

162 

students 

2.75 

(1.34) 

2.74 

(1.22) 

0.01 

0.01 

0 

.76 


Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. 

a For Pierce (1 984), the p-values presented here were reported in the original study. A correction for clustering was needed and resulted in a WWC-computed p-value of .50 for 
Lankton First-Year Algebra Test: Content Subscale 5-Graphic representation-, therefore, the WWC does not find the result to be statistically significant. The author also conducted an 
ANCOVA analysis that adjusted for student pretest scores on a measure of mathematical knowledge. However, due to missing data in the ANCOVA analysis, the WWC focuses on the 
unadjusted analysis reported in the study. Reported p-values are similar across the two analyses. Because pretest means and standard deviations were not reported, calculation of the 
program group mean using a difference-in-differences approach was not feasible. 
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Endnotes 

* Due to the 2015 restructuring of the Mathematics topic area from three areas (Elementary, Middle, and High School) to two areas (Pri- 
mary and Secondary Mathematics), this report is considered a new report rather than an updated report. The information in this report 
combines the research examined in the prior reports and presents the conclusions differently. 

1 The descriptive information for this program was obtained from a publicly available source: the publisher’s website (www.hmhco. 
com). The WWC requests publishers review the program description sections for accuracy from their perspective. The program 
description was provided to the publisher in August 2014, and the WWC incorporated feedback from the publisher. Further verification 
of the accuracy of the descriptive information for this program is beyond the scope of this review. 

2 The literature search reflects documents publicly available by November 2015. Two previous intervention reports were released in 
February 201 1 and February 2013 under the High School Math and Middle School Math topic areas, respectively. This report includes 
reviews of 33 studies that were not included in the prior reports. Of the additional studies, 32 were not within the scope of the review 
protocol for the Secondary Mathematics topic area, and one was within the scope of the review protocol but did not meet WWC group 
design standards. A complete list and disposition of all studies reviewed are provided in the references. 

The report includes reviews of all studies that previously met WWC group design standards with or without reservations and resulted 
in a revised disposition for five studies. 

Resendez and Azin (2006) and Resendez, Fahmy, and Manley (2005), in the Middle School Mathematics intervention report on Saxon 
Math issued in February 201 3, are excluded from this report because they fall outside of the scope of the Secondary Mathematics 
review protocol, since the majority of the samples in each study are using a primary mathematics intervention. An author query was 
conducted to request results for the subsample of students who are eligible for review under the Secondary Mathematics review pro- 
tocol, but no data to support such an analysis were received. 

Crawford and Raia (1986), also included in the Middle School Mathematics intervention report on Saxon Math issued in February 
2013, is excluded from this review because it is not eligible for review under the Secondary Mathematics review protocol. The sample 
uses a primary mathematics intervention ( Saxon Algebra Vz) and is eligible for review under the Primary Mathematics topic area. 

Pierce (1984) was previously listed as not eligible for review, given the study’s publication date based on an earlier evidence review 
protocol; however, the current Secondary Mathematics review protocol specifies a publication timeframe of 1983 or later, meaning this 
study is now eligible for review. 

In the previous report, Abrams (1989) was reported to meet WWC group design standards with reservations. However, an updated 
review determined the study does not establish equivalence of the analytic sample used to estimate impacts, and no additional infor- 
mation is available, so it does not meet WWC group design standards in this report. 

The studies in this report were reviewed using the Standards from the WWC Procedures and Standards Handbook (version 3.0), along 
with those described in the Secondary Mathematics topic area review protocol (version 3.1). The evidence presented in this report is 
based on available research. Findings and conclusions may change as new research becomes available. 

3 For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 1 9. These 
improvement index numbers show the average and range of student-level improvement indices for all findings across the studies. 

4 No studies examining the effectiveness of S axon Geometry, Saxon Algebra II, and Saxon Advanced Math fall within the scope of the 
Secondary Mathematics review protocol and meet WWC group design standards. Because no studies meet WWC group design stan- 
dards at this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness of Saxon 
Geometry, Saxon Algebra II, and Saxon Advanced Math on secondary students. Additional research that meets WWC standards is 
needed to determine the effectiveness or ineffectiveness of these courses. 

5 The “math-talented” designation is based on teacher recommendations and prior academic achievement. No information is provided 
on the specific thresholds that were used in delineating the math-talented criteria; however, all students in the sample scored at or 
above the 87th percentile on the California Achievement Test (CAT) Total Math Battery. 

6 The study author described the Orleans-Hanna Algebra Prognosis Test as the primary measure of student math achievement. The 
study also examined four study-generated criterion unit tests, not from the Orleans-Hanna Algebra Prognosis Test, designed to 
descriptively measure student understanding of algebraic components. However, the author did not provide information on the reliabil- 
ity or validity of these four tests. Accordingly, analyses based on these four unit tests were not considered in this version of the report. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2016, May). 

Secondary Mathematics intervention report: Saxon Math. Retrieved from http://whatworks.ed.gov 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 
of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 

The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 

The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Intervention 
Intervention report 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Single-case design 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 19. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 19. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 
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Glossary of Terms 


Standard deviation The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance Statistical significance is the probability that the difference between groups is a result of 

chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


Substantively important a substantively important finding is one that has an effect size of 0.25 or greater, regardless 

of statistical significance. 

Systematic review a review of existing literature on a topic that is identified and reviewed using explicit meth- 
ods. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their find- 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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